Generating a visual description tree based on a layout interruption record

ABSTRACT

A layout engine generates a visual layout tree for a visual description tree of a document whose content is to be rendered in one or more display areas. A visual description tree describes the content of a document along with formatting and logical arrangement the content. Visual description nodes of a visual description tree represent the organization of the constituent elements a document hierarchically. A visual layout tree defines the layout of a document within one or more display areas. Visual layout nodes of a visual layout tree represent the layout of the content of a document hierarchically. The layout engine receives as input a visual description tree and generates the corresponding visual layout tree. A rendering engine then inputs the visual layout tree and generates a graphics buffer containing the pixels representing the layout of the document as defined by the visual layout tree.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims the benefit of U.S. ProvisionalApplication Ser. No. 62/285,003 (previously U.S. Nonprovisionalapplication Ser. No. 14/481,875) entitled “GENERATING A VISUAL LAYOUTTREE FROM A VISUAL DESCRIPTION TREE REPRESENTATION OF A DOCUMENT,” filedon Sep. 9, 2014, which is incorporated herein in its entirety byreference.

BACKGROUND

Application programs, such as game applications, presentationapplications, e-reader applications, computer-aided design applications,and editing applications, are generating increasingly complex andsophisticated content to be presented on a display of a computingdevice. For example, a game application may generate content showingmovement of a firefighter through a burning building. A presentationapplication may generate content showing the unrolling of a scroll thatcontains a complex document with text and graphics in one portion of thedisplay and the waving of a flag in another portion of the display. Ane-reader application may generate content showing pages of a documentwith special effects such as animating the flipping of a page or manypages in rapid succession and displaying non-sequential pages of thedocument simultaneously in different display area. A computer-aideddesign application may generate content that animates movement of theinternals of a complex machine, such as a car engine or a mechanicalwatch. An editing application may generate content that applies specialeffects to the editing process such as showing a deleted page spiralinginto a trash can or animating the movement of an image from its currentlocation in the document to a new location.

The presenting of such complex and sophisticated content can requireconsiderable amount of computational resources. As a result, many lowcomputational power devices such as e-readers and smartphones cannotpresent such content. The more complex content might only be effectivelypresented on displays of high-end workstations with significantcomputational power. Even with such significant computational power, itmay not be possible to display complex content on a very large displaysuch as the size of a wall or billboard with tens of millions of pixels.It would be desirable to have more efficient techniques for presentingsuch content.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the display of an example document within a displayarea that is large enough to display the entire content of the exampledocument.

FIG. 2 illustrates the display of the example document within displayareas that are not large enough to display the entire content of theexample document.

FIG. 3 illustrates a display of a modified example document withindisplay areas that are not large enough to display the entire content ofthe example document.

FIG. 4 illustrates the organization of the example document in someembodiments.

FIG. 5 illustrates an example tree corresponding to the example documentin some embodiments.

FIG. 6 illustrates a visual layout tree for the display areas of FIG. 2.

FIG. 7 illustrates a visual layout tree for the display areas of FIG. 3.

FIG. 8 is a block diagram that illustrates the flow of content in someembodiments.

FIG. 9 is a block diagram that illustrates components of the layoutengine in some embodiments.

FIG. 10 is a flow diagram that illustrates the processing of a layoutcomponent of the controller of the layout engine in some embodiments.

FIG. 11 is a flow diagram that illustrates the processing of the layoutcomponent for a typical layout object in some embodiments.

FIG. 12 illustrates a visual layout node cache in some embodiments. Acache 1200 includes a cache store 1210 and a cache index 1220.

FIG. 13 is a flow diagram that illustrates the processing of a createvisual layout node component in some embodiments.

FIG. 14A is a block diagram illustrating the layout of a layoutinterruption record in some embodiments.

FIG. 14B illustrates organization of a layout handle.

FIG. 15 is a flow diagram that illustrates the processing of a layoutcomponent of a layout object to handle a layout interruption in someembodiments.

FIG. 16 illustrates the movement of a chess piece from a start locationto a destination location.

FIG. 17 is a block diagram that illustrates a change in location of aVLnode in support of animation in some embodiments.

FIG. 18 is a flow diagram that illustrates processing of a set upanimation pass of a VLtree in some embodiments.

FIG. 19 is a flow diagram that illustrates processing of an initializeanimation component in some embodiments.

FIG. 20 is a flow diagram that illustrates the processing of a getcoordinates component of a visual layout handle in some embodiments.

FIG. 21 is a diagram illustrating re-parenting of content in someembodiments.

FIG. 22 is a diagram that illustrates an example visual description treewith a named point.

FIG. 23 is a diagram that illustrates a fraction with a non-numericnumerator and denominator.

FIG. 24 is a diagram that illustrates an example VLtree generated forthe VDtree.

FIG. 25 is a diagram that illustrates the rendering of the fraction inthe display area.

FIG. 26 is a flow diagram that illustrates the processing of a partialregeneration component in some embodiments.

DETAILED DESCRIPTION

In some embodiments, a layout engine generates a visual layout tree fora visual description tree of a document whose content is to be renderedin one or more display areas. A visual description tree describes thecontent of a document along with formatting and logical arrangement thecontent. For example, a visual description tree may specify that achapter of a document includes paragraphs of text and an embedded image.Visual description nodes of a visual description tree represent theorganization of the constituent elements a document hierarchically. Forexample, a visual description tree may include a visual description noderepresenting a chapter with a child visual description node for eachconstituent element (e.g., paragraph or image) of the chapter. A visualdescription node representing a paragraph may have a child visualdescription node defining formatting (e.g., font type, zoom level, andcolor). A visual layout tree defines the layout of a document within oneor more display areas. For example, a visual layout tree may specifythat an image of a chapter is to be positioned at a specific location(e.g., coordinates (10,20)) within a display area. Visual layout nodesof a visual layout tree represent the layout of the content of adocument hierarchically. For example, a visual layout tree may include acontainer visual layout node representing a display area that containsone or more paragraphs of a chapter. The container visual layout nodemay have a child visual layout node for each paragraph that it containsand have positioning information indicating where the text of eachparagraph is to be positioned in the display area of the container. Thelayout engine receives as input a visual description tree and generatesthe corresponding visual layout tree. A rendering engine then inputs thevisual layout tree and generates a graphics buffer containing the pixelsrepresenting the layout of the document as defined by the visual layouttree.

In some embodiments, the layout engine generates a visual layout treefor a visual description tree whose content is to be displayed inmultiple display areas, referred to as being fragmented. For example, anapplication may specify to the layout engine that a document is to bedisplayed in two side-by-side display areas. The layout engine inputsthe visual description tree and an indication of the display areas andgenerates a visual layout tree with a subtree for each display area. Ifcontent associated with a visual description node is to be displayed inseparate display areas, the layout engine traverses the visualdescription tree for each display area and records where in thetraversal (e.g., a path down the visual description tree) the processingfor the current display area ended along with state informationdescribing the state at the time the processing ended, referred to as alayout interruption record. The layout engine then uses that layoutinterruption record for one display area to indicate the starting pointwithin the visual description tree that the layout for the next displayarea should begin. Initially, the layout engine may set a layoutinterruption record to indicate a beginning path of visual descriptionnodes that is the beginning of the document. For each display area thatis to contain content of the document, the layout engine processes thevisual description tree for visual description nodes starting withvisual description nodes along the path of the layout interruptionrecord, sets a current state for processing the visual description nodebased on a state indicated by the layout interruption record for thatvisual description node. The layout engine then populates the visuallayout tree with one or more visual layout nodes representing content ofthe visual description node to be included in the display area given thecurrent state. When processing of the content of the visual descriptionnode is interrupted for the display area, the layout engine adds to alayout interruption record for the next display area the then-currentstate of processing of the visual description node and an identifier ofthe visual description node. A path within the visual description treeis thus formed in the layout interruption record for the next displayarea of where the processing of the visual description tree wasinterrupted for the display area. The path defines where the processingfor the next display area is to start.

In some embodiments, the layout engine generates a visual layout treefor a visual description tree of a document based on cached information.The layout engine generates a first visual layout tree based on a firstvisual layout context by creating visual layout nodes of the firstvisual layout tree representing the document. The layout enginegenerates a visual layout node generated based on a node context the mayinclude the size of area (e.g., a layout restriction) in which thecontent of the visual layout node is to be laid out. The layout enginecaches the created visual layout nodes along with their node contexts.The layout engine may then generate a second visual layout tree based ona second visual layout context (e.g., portrait versus landscape). Thelayout engine generates visual layout nodes of the second visual layouttree by, when the node context for a visual layout node of the secondvisual layout matches the node context of a cached visual layout node,reusing the cached visual layout node for the second visual layout tree.When the node context for a visual layout node of the second visuallayout does not match the node context of a cached visual layout node,the layout engine creates a visual layout node for the second visuallayout tree for the new node context. In some embodiment, the cachedinformation may also include a layout interruption record so that visuallayout nodes are reused only when generated in the context of compatiblelayout interruption records.

In some embodiments, the layout engine generates a visual layout treefor a document that uses positioning pointers (e.g., visual layouthandles). The layout engine creates visual layout nodes of the visuallayout tree. For each visual layout node, the layout engine generates apositioning pointer for the visual layout node that links the visuallayout nodes to form the visual layout tree. Each positioning pointerhas a pointer to a child visual layout node and positioning informationfor positioning the child visual layout node within the context (e.g.,coordinate space) of a parent visual layout node so that the position ofthe child visual layout node within the context of the parent visuallayout node can be changed without having to recreate the visual layoutnode.

In some embodiments, the layout engine detects a change in positioninginformation of a visual layout node of a visual layout tree and may usethat change to animate the movement of content of the visual layout nodefrom the start location in a display area to the destination location inthe display area. The layout engine stores the visual layout tree withvisual layout nodes and visual layout handles that specify parent/childrelationship between visual layout nodes. A visual layout handle has areference to a child visual layout node and to its parent visual layoutnode along with positioning information indicating position of the childvisual layout node within the coordinate space of its parent visuallayout node. Each child visual layout node has a back reference to thevisual layout handle that references it. The layout engine generates amodified visual layout tree in which a moved child visual layout nodehas been moved from a start location within the coordinate space of itsparent visual layout node to a destination location within thecoordinate space of its parent visual layout node. The layout enginegenerates a new visual layout handle that has a reference to the movedchild visual layout node and the same parent visual layout node alongwith new positioning information specifying the destination location.Upon detecting that the back reference of the moved child visual layoutnode references a visual layout handle different from the newlygenerated visual layout handle, the layout engine indicates that themoved visual layout node has been moved. Since both visual layouthandles have a reference to the same parent visual layout node, thelayout engine can detect that the start location and the destinationlocation are within the coordinate space of the same parent visuallayout node.

In some embodiments, the layout engine animates the movement of thecontent by initializing a storyboard to generate a path of coordinatesfrom the start location to the destination location and setting the newvisual layout handle to indicate animation is in progress. When therendering engine requests positioning information for the new visuallayout handle, the rendering engine accesses the storyboard to retrievecoordinates for a point along a path from the start location to thedestination location. Thus, as the rendering engine successivelygenerates frames to be displayed, different positioning information forthe visual layout node is generated so that the rendering engine canrender the content at successively different locations within a graphicsbuffer.

In some embodiments, the layout engine generates a visual layout treethat is virtualized for use in rendering content of a document. Thelayout engine receives an anchor point within the document. The anchorpoint identifies a visual description node within the visual descriptiontree and a named point within content of the identified visualdescription node. A named point may be, for example, on a rectangle ofthe measured size (e.g., bounding rectangle of the content), such as thelower right corner of the measured size. A name point may also bespecific to the type of content. For example, content representing theface of a clock may have a named point for each of the 12 hours. Thelayout engine also receives a target location within a display area inwhich content associated with the anchor point is to be positioned. Aperson reading the document using an e-reader application may select afigure of a clock from an index of figure, and the e-reader applicationmay direct the layout engine to generate a visual layout tree forrendering a portion of the document on a display area with the 10o'clock named point at a certain position in the display area. Thelayout engine traverses the visual description tree along a path fromthe root of the visual description tree to the identified visualdescription node. While traversing the layout engine generates visuallayout nodes for the visual description nodes along the path and avisual layout node for the identified visual description node, whichrepresents content of the identified visual description node. The layoutengine then generates positioning information for the visual layoutnodes along the path from the root visual layout node to the visuallayout node generated for the identified visual description node so thatthe name point of the content is positioned at the target location inthe display area. When the content of the identified visual descriptionnode does not fill the display area, the layout engine generates visuallayout nodes for visual description nodes adjacent to the identifiedvisual description node to fill the display area with content withouthaving to generate visual layout nodes for the entire document.

In some embodiments, the content of a document may be represented as athree-dimensional object. For example, if the document contains atriangular prism, then the visual description tree for the document mayinclude a visual description node specifying a triangular prism. Thepositioning information for the visual layout node may specify athree-dimensional rotation of the triangular prism as a pitch, roll, andyaw and three-dimensional translation of the triangular prism. When therendering engine renders the triangular prism, it applies the specifiedrotation and generates a projection onto a two-dimensional surface. Ifthe document includes text to be rendered on three of the sides of thetriangular prism, then the visual description tree may include visualdescription nodes for three paragraphs representing text for the sides.When generating the visual layout tree, the layout engine generates avisual layout node for the triangular prism and visual layout nodesindicating that three paragraphs are to be applied as two-dimensionaltextures to the sides of the triangular prism. The triangular prism maybe represented in the visual description tree with a visual descriptionnode for the triangular prism with a child visual description node foreach of the three sides and two ends. The visual description nodescorresponding to each of the three sides have a child visual descriptionnode representing the paragraph that is to appear on the side. Therendering engine renders the triangular prism with the correspondingtextures based on the visible sides of the prism in accordance with thepitch, roll, and yaw. In some embodiments, the content of a document mayspecify the three-dimensional rotation of a two-dimensional object. Forexample, a paragraph containing text may have positioning informationindicating a pitch of 45 degrees (i.e., rotation around the x-axis). Insuch a case, the rendering engine renders the text of the paragraph asif placed on a surface at an angle of 45 degrees to the display area.

In some embodiments, a visual layout tree may be used to supportthree-dimensional hit testing to identify content corresponding to aspecified location on a display. For example, when editing a documentthat contains a triangular prism with text on the sides of the prism, auser may point to a location on the display within a bounding rectangleof a character of the text. The corresponding face of the triangularprism that contains the selected character and the position of thepointed to location within the face (or text of the face) can bedetermined from the positioning information including rotation of theprism.

FIG. 1 illustrates the display of an example document within a displayarea that is large enough to display the entire content of the exampledocument. The example document contains paragraphs 1, 2, and 3.Paragraph 1 contains text 1.1, image 1.2, and text 1.3; paragraph 2contains text 2.1 and image 2.2; and paragraph 3 contains text 3.1. Thedisplay device 100 includes a display area 110, which may correspond tothe entire displayable area of the display device (e.g., smart phonescreen) or a portion of the display device (e.g., window for displayingthe example document). Because of the size of the display area 110,paragraphs 1, 2, and 3 (i.e., the entire document) are displayed at thesame time.

FIG. 2 illustrates the display of the example document within displayareas that are not large enough to display the entire content of theexample document. The display device 200 includes display areas 210 and220. The display areas may correspond to a scrollable page view for theexample document where the pages scroll horizontally. A first portion ofthe example document is displayed in display area 210, a second portionis displayed in display area 220, and a third portion is not displayed.Since paragraph 1 cannot fit into display area 210, a first portion ofparagraph 1 is displayed in display area 210, and a remaining secondportion of paragraph 1 is displayed in display area 220. Since text 1.1cannot fit into display area 210, a first portion of text 1.1, referredto as text 1.1.1, is displayed in display area 210. A remaining secondportion of text 1.1, referred to as text 1.1.2, is displayed in displayarea 220 along with the remainder of paragraph 1. Although display areas210 and 220 are illustrated as being displayed simultaneously on thedisplay device 220, a display area may represent the entire area of asmall display device on which only a portion of the example document canbe displayed at a time (e.g., a scrollable view with only one displayarea).

FIG. 3 illustrates a display of a modified example document withindisplay areas that are not large enough to display the entire content ofthe example document. The example document has been modified so thatimage 1.2 is rotated 90 degrees. The display device 300 includes displayareas 310 and 320. The display area 310 displays content that isidentical to the content of display area 210. The display area 320displays text 1.1.2 and the rotated image 1.2. Because image 1.2 hasbeen rotated, text 1.3 can no longer fit into the same display area astext 1.1.2 and image 1.2. If the display device also included displayarea 330, then text 1.3 would be laid out as part of display area 330,as indicated by the dashed lines.

FIG. 4 illustrates the organization of the example document in someembodiments. The documents that are input by the layout engine arestored as a tree data structure, referred to as a visual descriptiontree or VDtree, that contains the content and formatting of thedocument. The VDtree 400 includes nodes 401-412, referred to as VDnodesto indicate that they are nodes of a VDtree. Each VDnode has anassociated type of either a document organizational unit or formattingassociated with an organizational unit. The types of organizationalunits may include portal, horizontal list, vertical list, chapter,index, paragraph, text, image, graph, map, footnotes, and so on. Theformatting type indicates that the VDnode contains formattinginformation for its parent organizational unit. VDnode 401 is of typeroot and represents the entire document. VDnode 402 is of type portaland represents that its child nodes are to be laid out in a singleportal. VDnode 403 is of type horizontal list of its child nodes. VDnode404 is of type page and represents that its child nodes (i.e., that arenon-formatting nodes) are to be laid out as pages. VDnodes 405 and 406are of type paragraph and represent that their child nodes are to belaid out as separate paragraphs. VDnode 405 corresponds to paragraph 1of the example document. VDnodes 407 and 409 are of type text andrepresent text of the paragraph represented by VDnode 405. If the textof a paragraph has different formatting characteristics (e.g.,highlighting), then the VDnode for the paragraph may have a child VDnodefor each span of text with the same formatting. VDnode 408 is of typeimage and represents an image within the paragraph represented by VDnode405. VDnodes 407, 408, and 409 may represent the text by containing thetext, a reference to the text, or some other mapping to thecorresponding text. VDnodes 410, 411, and 412 are of type formatting andrepresent formatting that applies to their parent nodes. For example,VDnode 410 may indicate that each page is to have a certain type ofborder or margin width. VDnode 411 may indicate indentation of the firstline, line spacing, font type, and so on for the paragraph representedby VDnode 405. VDnode 412 may indicate font type for the textrepresented by VDnode 407, which may override the font type indicated byVDnode 411.

The layout engine inputs a VDtree and generates a visual layout tree,referred to as a VLtree, that contains content of the document withlayout information for displaying the content within the display areas.A rendering engine then inputs the VLtree and renders the content of theVLtree in accordance with the layout information of the VLtree. A VDtreeconceptually describes how a document should be layout in general. AVLtree, in contrast, describes the actual layout of the content to aspecific display area. The VLtree includes visual layout nodes, referredto as VLnodes to indicate that they are nodes of a VLtree. The VLnodesspecifies the coordinates of where content is to be located. A parentVLnode has a coordinate space (e.g., 1024 by 1024) and specifies wherethe content of each child VLnode is to be positioned in that coordinatesspace. Each child VLnode also has its own coordinate space (e.g., 256 by256), which in turn specified the content of each of its child VLnodesis to be position. The layout engine performs mappings between thecoordinate spaces.

FIG. 5 illustrates an example tree corresponding to the example documentin some embodiments. The layout engine generates VLtree 500 forrendering in the display area 110 the example document as represented byVDtree 400. The VLtree 500 contains VLnodes may be of type container,graphics, graphical object (“gob”), and so on. A VLnode of typecontainer represents a grouping of child VLnodes, a VLnode of typegraphics represents a type of content (e.g., text or image), and aVLnode of type graphical object represents the content for its parentVLnode. VLnodes 501-506 are container nodes corresponding to VDnodes401-406. VLnodes 507 and 510 correspond to VDnode 407, VLnodes 508 and511 correspond to VDnode 408, and VLnodes 509 and 512 correspond toVDnode 408. VLnodes 507-509 are graphics nodes for text or imagecontent. VLnodes 510-512 represent content of the document. Since VLnode505 corresponds to the paragraph VDnode 405, VLnodes 510-512 representthe content of that paragraph (i.e., paragraph 1). In some embodiments,the VLnodes 507-509 may be merged with their child VLnodes. In such acase, the VLnodes 507-509 may contain the content or a reference to thecontent.

The directed links between the VLnodes of a VLtree are represented byobjects referred to as visual layout handles. A visual layout handlecontains a reference to a child VLnode along with some formattinginformation such as positioning information and color information forthat child VLnode. FIG. 14B illustrates organization of a visual layouthandle. The positioning information may include the x, y, and zcoordinates for placing the content represented by the VLnode within thecoordinate space of the parent VLnode. The positioning information mayalso include orientation information (e.g., rotate 90 degrees) on how toorient the content, scaling information (e.g., 150%) on how to scale thecontent within the coordinate space of the parent VLnode, and z-orderrelated information. For example, the visual layout handle pointing fromVLnode 504 to VLnode 505 may indicate that paragraph 1 is to bepositioned at coordinates (120, 10) within the coordinate space ofVLnode 504 and rotated zero degrees.

When a document is to be rendered in multiple display areas, the layoutengine may generate a VLtree with a subtree for each display area thatcontains the VLnodes for rendering a portion of the document in thatdisplay area. FIG. 6 illustrates a visual layout tree for the displayareas of FIG. 2. The VLtree 600 includes VLnodes 601-604 correspondingto VLnodes 501-504. VLnode 605 is the root node of a subtreecorresponding to the content of display area 210, and VLnode 606 is theroot node of a subtree corresponding to the content of display area 220.Since paragraph 1 does not fit into display area 210, the content forVLnode 605 includes only the content of text 1.1.1, and the content forVLnode 606 includes the remaining portions of paragraph 1.

FIG. 7 illustrates a visual layout tree for the display areas of FIG. 3.In this example, image 1.2 is rotated 90 degrees. The VLtree 700includes VLnodes 701-704 corresponding to VLnodes 501-504. VLnode 705 isthe root node of a subtree corresponding to the content of display area310, VLnode 706 is the root node of a subtree corresponding to thecontent of display area 320, and VLnode 707 is the root node of asubtree corresponding to the content of display area 330, which is notdisplayed in FIG. 3. VLnodes 710 and 714 may have content that isidentical to that of VLnodes 610 and 614 with the rotation of 90 degreesbeing represented by the visual layout handle from VLnode 706 to VLnode710. When the rendering engine processes VLnode 710, it rotates thecontent as indicated by the visual layout handle.

FIG. 8 is a block diagram that illustrates the flow of content in someembodiments. The application 801 generates a VDtree representing thecontent to be rendered. The layout engine 802 inputs the VDtree andgenerates the corresponding VLtree. For each VDnode, the layout engineinstantiates a visual layout objects of the type of the VDnode. Forexample, the layout engine would instantiate visual layout object oftype paragraph for a VDnode representing a paragraph. The layout enginethen invokes the layout component of the visual layout objects togenerate the corresponding VLnodes. When a new type of VDnode is definedor the behavior of an existing layout object is changed, a new visuallayout object that implements the desired behavior can be provide to thelayout engine. The layout engine may recursively invoke the layoutcomponents of the visual layout objects in a traversal of a VDtree. Thelayout engine initially invokes the layout component of a visual layoutobject for the root VDnode of the VDtree. That layout component therequests the layout engine to invoke layout components of visual layoutobjects for the child VDnodes of the root VNnode. When invoking a layoutcomponent on behalf of a parent VDnode, the layout engine provides arectangle corresponding to the size of the area within a display area.The layout component generates the corresponding VLnodes and returns tothe layout engine the amount of the rectangle that it used to layout itscontent. The layout engine returns size of the remaining rectangle tothe layout component for the parent VDnode so that that layout componentcan determine whether to invoke layout components corresponding to otherchild VDnodes or to return that size to the layout component of itsparent VDnode. The rendering engine 803 inputs the VLtree and rendersthe corresponding content to a graphics buffer for output to display804.

FIG. 9 is a block diagram that illustrates components of the layoutengine in some embodiments. The layout engine 900 includes a controllerlayout component 901, Lobject layout components 902, a partialregeneration component 903, a create VLnode component 904, a setupanimation component 905, an initialize animation component 906, and aVLnode cache 907. The controller layout component is invoked to generatea VLtree for a VDtree. Each Lobject layout component is invoked togenerate the VLnodes corresponding to a VLnode. The partial regenerationcomponent is invoked to regenerate a portion of a VLtree based onannotations on the VDnodes indicating which VDnodes have been modified.The create VLnode component creates a VLnode or returns a reference to acached VLnode. The setup animation component and the initializeanimation component control the animation of the rendering of a VLtree.

The computing devices and systems on which the layout engine may beimplemented may include a central processing unit, input devices, outputdevices (e.g., display devices and speakers), storage devices (e.g.,memory and disk drives), network interfaces, graphics processing units,accelerometers, cellular radio link interfaces, global positioningsystem devices, and so on. The input devices may include keyboards,pointing devices, touchscreens, gesture recognition devices (e.g., forair gestures), head and eye tracking devices, microphones for voicerecognition, and so on. The computing devices may include desktopcomputers, laptops, tablets, e-readers, personal digital assistants,smartphones, gaming devices, servers, and computer systems such asmassively parallel systems. The computing devices may accesscomputer-readable media that includes computer-readable storage mediaand data transmission media. The computer-readable storage media aretangible storage means that do not include a transitory, propagatingsignal. Examples of computer-readable storage media include memory suchas primary memory, cache memory, and secondary memory (e.g., DVD) andinclude other storage means. The computer-readable storage media mayhave recorded upon or may be encoded with computer-executableinstructions or logic that implements the layout engine. The datatransmission media is used for transmitting data via transitory,propagating signals or carrier waves (e.g., electromagnetism) via awired or wireless connection.

The layout engine may be described in the general context ofcomputer-executable instructions, such as program modules andcomponents, executed by one or more computers, processors, or otherdevices. Generally, program modules or components include routines,programs, objects, data structures, and so on that perform particulartasks or implement particular data types. Typically, the functionalityof the program modules may be combined or distributed as desired invarious embodiments. Aspects of the layout engine may be implemented inhardware using, for example, an application-specific integrated circuit(“ASIC”).

FIG. 10 is a flow diagram that illustrates the processing of a layoutcomponent of the controller of the layout engine in some embodiments.The layout component 1000 is invoked when the VLtree for a VDtree is tobe generated. For example, an application that modifies a document andthus the VDtree may invoke the component to regenerate the VLtree forrendering the modified document. The layout component has access to theVDtree for the document and a display area. In this illustratedprocessing, the component assumes that the display area is large enoughto contain the entire document. In block 1001, the component retrievesthe root VDnode. In block 1002, the component creates a Lobject for theroot VDnode. The Lobjects implement the behavior for generating theVLnodes for a certain type of VDnode. For example, a Lobject for aparagraph implements behavior specific to a paragraph. In block 1003,the component sets the Lobject to point to the root VDnode. Each Lobjectwill eventually point to its corresponding VDnode and VLnode. In block1004, the component invokes the layout component of the created Lobjectand completes.

FIG. 11 is a flow diagram that illustrates the processing of the layoutcomponent for a typical layout object in some embodiments. The component1100 generates VLnodes for the subtree corresponding to the VDnode ofthis Lobject. In blocks 1101-1104, the component creates a VLnode andlinks the VLnode into the VLtree. In block 1101, the component creates aVLnode specific to the VDnode of this Lobject. For example, if theVDnode is of type text, then the component creates a graphics VLnodethat represents text. In block 1102, the component sets this layoutobject to reference the VLnode so that this layout object referencesboth the VDnode and its corresponding VLnode. In block 1103, thecomponent creates a visual layout handle for the VLnode. In block 1104,the component adds the visual layout handle to the parent VLnode exceptwhen that VLnode is the root of the VLtree. In blocks 1105-1110, thecomponent loops processing each child of the VDnode of this Lobject. Inblock 1105, the component selects the next child VDnode. In decisionblock 1106, if all the child VDnodes have already been selected, thenthe component completes, else the component continues at block 1107. Inblock 1107, the component creates a Lobject of the type of the childVDnode. In block 1108, the component sets the created Lobject to pointto the selected VDnode. In block 1109, the component performstype-specific processing for the Lobject. For example, if the Lobjectrepresents a text VDnode, then the component may generate the glyphs forthe text, add a VLnode of type graphical object, and add to the VLnode areference to the glyphs. When generating the glyphs, the componentfactors in formatting information of the VDnode and its ancestorVDnodes. The component may invoke standard functions of an operatingsystem to generate the glyphs. In block 1110, the component invokes thelayout component of the newly created Lobject to generate the portion ofthe VLtree corresponding to the selected child VDnode and then loops toblock 1106 to select the next child VDnode. Although the layoutcomponent is illustrated as directly invoking the layout components ofits child Lobjects, the layout component may instead request the layoutengine to invoke the layout components of its child Lobjects on itsbehalf. Such a request allows the layout engine to take control andprovider overall control of the layout process and perform processingthat would otherwise need to be performed by each type of Lobject. Forexample, the layout engine may update the positioning information in thevisual layout handles.

FIG. 12 illustrates a visual layout node cache in some embodiments. Acache 1200 includes a cache store 1210 and a cache index 1220. The cachestore contains entries corresponding to VLnodes that have been created.When the VLtree is regenerated (e.g., when a display is rotated fromportrait view to landscape view), portions of the newly generated VLtreemay be the same as a prior VLtree. In such a case, the layout engine canreuse those VLnodes of the cache without having to regenerate them. Thelayout engine creates the VLnode within the cache store. The layoutengine creates a VLnode based on a current state of the layout processwhen the layout component that creates the VLnode is invoked. Forexample, the current state may be a combination of informationidentifying the corresponding VDnode, a demand rectangle, layout context(e.g., formatting information), a layout interruption record, and so on.The layout engine assigns a cache key to each cached VLnode thatidentifies the current state at the time the VLnode was created. Whenthe VLtree is regenerated, the cache can be searched for a VLnode withthe same state. If such a VLnode is found, it can be reused. Otherwise,the layout engine generates a new VLnode in the cache. To facilitatesearching for a VLnode with a matching cache key, the layout enginestores the cache keys in a cache index that maps the cache keys to theircorresponding VLnodes in the cache store (or cache storage). The cacheindex may be organized using a variety of data structures such as a hashtable, tree data structure, and so on.

FIG. 13 is a flow diagram that illustrates the processing of a createvisual layout node component in some embodiments. The component 1300checks the cache to see whether a VLnode already has been created forthe same state. If so, the component reuses the VLnode, else thecomponent creates a VLnode in the cache. In block 1301, the componentcreates a cache key based on the current state of the layout processing.In block 1302, the component checks the cache for a matching cache key.In decision block 1303, if a matching cache key is found, the componentreturns the corresponding pointer to the VLnode in the cache, else thecomponent continues at block 1304. In block 1304, the component createsa VLnode in the cache. In block 1305, the component updates the cacheindex to map the cache key to the pointer to the newly created VLnodeobject and then returns the pointer.

Layout Interruption

FIG. 14A is a block diagram illustrating the layout of a layoutinterruption record in some embodiments. The layout interruption record1400 contains the current state of a path in the VDtree that cannotcompletely fit within the current display area. Each Lobject returns alayout stop record indicating what portion of its VDnode was laid out.The layout stop record may indicate “end” when the VDnode has beencompletely laid out or may be a layout interruption record thatindicates the current state at the time the layout stopped. When aparent Lobject receives a stop layout interruption record indicatingthat a child VDnode has not been completely laid out, the parent Lobjectadds an identifier of that child VDnode and its own current state to thestop layout interruption record and returns the layout stop record toits parent Lobject. The layout interruption record includes child state1401, child identifier 1402, non-child state 1403, parent identifier1404, and non-child state 1405. The child state 1401 was returned to aparent Lobject, which added the child identifier 1402 and the currentstate, non-child state 1403, of the parent Lobject. The parent state1406 was returned to the grandparent Lobject, which added the parentidentifier 1404 and the current state, non-child state 1405, of thegrandparent. The grandparent state 1407 was then returned to thegreat-grandparent Lobject and so on until the Lobject corresponding tothe root node was encountered. In some embodiments, upon receiving anindication that the layout of a child VDnode has not been completed, aparent Lobject may continue to process its other child VDnodes. In sucha case, each child Lobject may return its own current state, and theparent Lobject generates a layout interruption record that includes thechild identifier and state of each child Lobject whose layout was notcompleted.

When a Lobject is invoked, its parent Lobject passes a layout startrecord indicating where in the corresponding VDnode the layout shouldstart. Typically, the layout start record would indicate the beginningof the VDnode. If the layout start record is a layout interruptionrecord, the invoked Lobject sets its state to that of the layoutinterruption record and continues to lay out the VDnode where it lastleft off.

FIG. 15 is a flow diagram that illustrates the processing of a layoutcomponent of a layout object to handle a layout interruption in someembodiments. The flow diagram illustrates the processing of the layoutcomponent just for handling a layout start record, which in this exampleis a layout interruption record. The 1500 component is passed a layoutinterruption record and returns a layout stop record, which is a layoutinterruption record if the layout was interrupted. In blocks 1501-1509,the component loops selecting each child VDnode. In block 1501, thecomponent selects the next child VDnode. In decision block 1502, if allthe child VDnodes have already been selected, then the componentcompletes, else the component continues at block 1503. In decision block1503, if the identifier of the VDnode is in the layout start record, thecomponent continues at block 1504, else the component continues at block1506. In block 1504, the component extracts the state for the VDnodefrom the layout start record. In block 1505, the component sets thelayout start record for the child VDnode to the extracted state. Inblock 1506, the component sets the layout start record for the childVDnode to indicate to start at the beginning. In block 1507, thecomponent invokes the layout component of the child VDnode passing thelayout start record for the child VDnode. In decision block 1508, if theinvoked layout component returns a layout stop record indicating thatthe layout stopped at the end of the child VDnode, then the componentloops to block 1501 to select the next child VDnode, else the componentcontinues at block 1509. In block 1509, the component adds to the layoutstop record the identifier of the child VDnode and optionally any stateinformation that the component may need to resume layout of the childVDnode. The component then loops to block 1501 to select the next childVDnode.

Animation

In some embodiments, the layout engine provides for automatic animationin the rendering of movement from a first version of a document to asecond version of the document. For example, a chess game applicationmay generate different versions of the document corresponding to thecurrent position of the chess pieces on a chessboard. When a playerindicates to move a chess piece from one location to another, the layoutengine allows that movement to be automatically animated to show thegradual movement of that chess piece to the other location rather thansimply having the chess piece abruptly show up at the other location.When the application generates the second version of the document, thelayout engine detects the changed location of the chess piece,initializes a storyboard for generating a path from the current locationto the destination location, and generates a visual layout handle forthe chess piece that references the storyboard. During rendering, whenthat visual layout handle is accessed by the rendering engine to renderthe next frame, the visual layout handle provides positioninginformation based on the current state of the storyboard, that is, aposition along the path from the current location to the destinationlocation. In this way, the layout engine can provide animation withoutthe application having to regenerate the visual layout tree for eachframe of the animation.

FIG. 16 illustrates the movement of a chess piece from a start locationto a destination location. The chess game application may generate aVDtree for the display area 1610 that includes a rook at location 1611and a knight at start location 1612. When a chess player indicates tomove the knight to a different location, the chess game application maygenerate a visual description tree for the display area 1630. Thedisplay area 1630 indicates that the rook is at the same location 1631,but that the knight has moved to destination location 1633. The layoutengine generates a visual layout handle for the VLnode representing theknight that generates positioning information for each frame to effectthe animation of the movement of the knight from start location 1612 tothe destination location 1633. The display area 1620 illustrates themovement of the knight during the animation. The layout engine defines apath from location 1622 to location 1623 so that the rendering enginecan animate the movement the knight as it renders successive frames intothe graphics buffer.

The layout engine detects when a VLnode has been moved from a startlocation to a destination location based on inspection of the VLtreeduring an intermediate pass when generating the VLtree with the VLnodeat the destination location. FIG. 17 is a block diagram that illustratesa change in location of a VLnode in support of animation in someembodiments. Subtree 1710 represents a portion of a VLtree for thedisplay area 1610 and includes VLnodes 1711, 1712, and 1713. VLnode 1711is of type container and contains each of the chess pieces. VLnode 1712corresponds to the rook, and VLnode 1713 corresponds to the knight.VLnode 1711 points to visual layout handle (“VLh”) 1714 and a visuallayout handle 1715. The visual layout handle 1714 points to and containspositioning information for VLnode 1712 and points to VLnode 1711. Thevisual layout handle 1715 points to and contains the positioninginformation for VLnode 1713 and points to VLnode 1711. VLnodes 1712 and1713 contain back pointers to the visual layout handles that point tothem. Subtree 1720 represents a portion of the VLtree for display area1620 and includes VLnodes 1721, 1722, and 1723, which correspond toVLnodes 1711, 1712, and 1713, respectively. Because the positioninginformation for the rook has not changed, VLnode 1721 points to the samevisual layout handle 1714 as in subtree 1710. VLnode 1721, however,points a visual layout handle 1725 that is different from the visuallayout handle 1715 of subtree 1710. Visual layout handle 1725 containspositioning information for the destination location 1633. After avisual layout handle for a VLnode is generated, the layout engine checksfor a change in location and sets the back pointer to the visual layouthandle. The layout engine detects that the back pointer for VLnode 1723points to visual layout handle 1715, which is different from the visuallayout handle 1725 that is currently pointing to it, indicating a changeof position for VLnode 1723. If animation is enabled, the layout engineinitializes a storyboard to generate a path from start location 1612indicated in visual layout handle 1715 to the destination location 1622indicated by visual layout handle 1725. The layout engine then sets theback pointer for VLnode 1723 to point to visual layout handle 1725. Assubtree 1720 is repeatedly rendered for a sequence of frames, visuallayout handle 1725 returns different positioning information inaccordance with the path specified by the storyboard to effect theanimation.

FIG. 18 is a flow diagram that illustrates processing of a set upanimation pass of a VLtree in some embodiments. The component 1800 ispassed a parent visual layout handle for a VLnode and determines whetheranimation should be initialized for that VLnode and recursively invokesthe component for child VLnodes of that VLnode. In block 1801, thecomponent retrieves the back visual layout handle for that VLnode. Indecision block 1802, if the back visual layout handle is the same as theparent visual layout handle, then the VLnode has not changed itslocation and the component continues at block 1805, else the VLnode haschanged its location and the component continues at block 1803. In block1803, the component invokes an initialize animation component passing anindication of the parent visual layout handle and the back visual layouthandle. In block 1804, the component sets the back visual layout handleto point to the correct visual layout handle. In blocks 1805-1807, thecomponent loops recursively invoking the set up animation component foreach child VLnode. In block 1805, the component selects the next childVLnode. In decision block 1806, if all the child nodes have already beenselected, then the component completes, else the component continues atblock 1807. In block 1807, the component invokes the set up animationcomponent passing an indication of the child visual layout handle andthen loops to block 1805 to select the next child VLnode.

FIG. 19 is a flow diagram that illustrates processing of an initializeanimation component in some embodiments. The component 1900 is passed anindication of a parent visual layout handle and a back visual layouthandle and initializes animation from the location of the back visuallayout handle to the location of the parent visual layout handle. Inblock 1901, the component creates a storyboard. In block 1902, thecomponent sets the start location of the storyboard based on the backvisual layout handle. In block 1903, the component sets the destinationlocation based on the parent visual layout handle. In block 1904, thecomponent sets a path to follow when moving from the start location tothe destination location. In block 1905, the component sets the numberof intervals or frames that will be generated to show the animationalong the path from the start location to the destination location. Inblock 1906, the component sets an animation pointer in the parent visuallayout handle to point to the created storyboard and indicate thatanimation is enabled for the parent visual layout handle. The componentthen completes.

FIG. 20 is a flow diagram that illustrates the processing of a getcoordinates component of a visual layout handle in some embodiments. Thecomponent 2000 is invoked by the rendering engine to retrieve thecoordinates associated with a visual layout handle. In decision block2001, if the visual layout handle has its animation pointer set, thenthe component continues at block 2003, else the component continues atblock 2002. In block 2002, the component retrieves the staticcoordinates of the visual layout handle and then returns thosecoordinates. In block 2003, the component invokes a get next coordinatescomponent of the storyboard for the visual layout handle to retrieve thenext coordinates along the path from the start location to thedestination location and then returns those coordinates.

The animation of a change in position from a start location to adestination location is somewhat more complex when the VLnode thatchanges position has a different parent. For example, when a chess pieceis captured, the chess game application may move the chess piece from alocation on the chessboard to a location off the chessboard in acaptured pieces area. The corresponding VLtree may include a containerfor the pieces currently located within the chessboard and a containerfor the pieces that are currently captured. The moving of content fromone container to another container is referred to as “re-parenting.”Since content can be displayed only within the area of its parentcontainer, the animation of movement from one parent container to a newparent container cannot readily be represented by placing the content ineither the old parent container or the new parent container.

When the layout engine detects that a VLnode in the destinationcontainer has been re-parented, the layout engine sets a flag in thevisual layout handle for that VLnode indicating that it has beenre-parented. The layout engine also adds a temporary VLnode to theclosest common ancestor VLnode of the start container and thedestination container. The layout engine then initializes a storyboardto support the animation of the temporary VLnode from the start locationto the destination location as described above. Because the coordinatespace of the closest common ancestor VLnode encompasses the coordinatespaces of both the start container and the destination container, thevisual layout handle can return a coordinate path that starts at thestart location within the start container and ends at the destinationlocation in the destination container. When the rendering engine detectsthat the VLnode within the destination container has been re-parented,it can suppress generating any content for that VLnode. At the end ofthe animation, the get coordinates component of the visual layout handleof the temporary node can remove the temporary VLnode and reset there-parenting flag in the VLnode of the destination container so that thenext rendering will be based on the VLnode being at its destinationlocation within the destination container. In some embodiments, thelayout engine may temporarily augment the VDtree to effect the animationby adding a temporary VDnode for the VDnode that has moved as a child ofits original parent VDnode. During the animation process, the layoutengine may successively move of the temporary VDnode up the path ofancestor VDnodes to the closest common ancestor and down the path ofdescendant VDnodes to it final location. In this way, as the animationprogresses the VLtree can be regenerated based on characteristics of thecurrent parent VDnode of the temporary VDnode.

FIG. 21 is a diagram illustrating re-parenting of content in someembodiments. Subtree 2110 of a VLtree represents chess pieces within achessboard container. Subtree 2110 includes container VLnode 2111,chessboard container VLnode 2112, captured pieces container VLnode 2113,rook VLnode 2114, and knight VLnode 2115. The chessboard containerVLnode 2112 contains a visual layout handle 2116 pointing to the rookVLnode 2114 and a visual layout handle 2117 pointing to the knightVLnode 2115. The rook VLnode 2114 contains a back pointer to visuallayout 2116, and knight VLnode 2115 contains a back pointer to visuallayout handle 2117. Subtree 2120 includes container VLnode 2121,chessboard container VLnode 2122, captured pieces container VLnode 2123,rook VLnode 2124, and knight VLnode 2125. Subtree 2120 illustrates thatthe knight VLnode 2125 has been moved from being a child of chessboardcontainer VLnode 2112 to being a child of captured pieces container2123. Captured pieces container 2123 includes visual layout handle 2127that points to knight VLnode 2125. After initial processing, the backpointer of the knight VLnode 2125 points to the visual layout handle2117, which in turn points to chessboard container VLnode 2112. Thelayout engine detects not only that the back pointer of knight VLnode2125 no longer points to its current visual layout handle 2127 but alsothat the visual label handle 2117 points to a different parent VLnode asan indication of re-parenting. When the layout engine detects suchre-parenting, the layout engine creates a temporary knight VLnode 2126as a child VLnode of the closest common container, which is containerVLnode 2121, and initializes a storyboard for generating a coordinatepath from the start location within chessboard container VLnode 2122 tothe destination location.

Virtualization

In some embodiments, the layout engine generates only as much of theVLtree as is needed to fill the display area. For example, a userreviewing a large document may review an index of figures within thedocument and request that a certain figure be displayed. In such a case,the layout engine may generate a VLtree that includes a VLnode for thatfigure and VLnodes for enough text to fill the display area above andbelow or to the left and right of the figure. Once the user selects thefigure, the application may direct the layout engine to generate theVLtree with the center of that figure positioned at a certain positionin the display area. The portion of the VDtree (e.g., center of thefigure) that is to be positioned at the certain position of the displayis referred to as the “anchor point” for the VLtree. An anchor pointidentifies an anchor VDnode and anchor coordinates within the coordinatespace of the anchor VDnode. The process of generating only as much ofthe VLtree as is needed to fill a display area is referred to as“virtualization” because the VLtree for the entire document does notexist and is only virtually generated. In some embodiments, the VDtreemay also be virtualized so that the application generates only thoseportions of the VDtree to support generating of the VLtree needed torender into a display area. When an Lobject for a VDnode attempts toretrieve a child VDnode, the layout engine may notify the application.The application can then generate the child VDnodes for return to theLobject. Similarly, when the Lobject for a child VDnodes attempts toretrieve its child VDnode, the layout engine again notifies theapplication, which generates child VDnodes of the child VDnodes.

To facilitate the positioning of content, the layout engine may allowvarious points within content to be named, referred to as “namedpoints.” The layout engine may recognize a set of named points relatingto the bounding rectangle or measured size of content. For example, thenamed points may include upper left, upper center, upper right, middleleft, center, middle right, lower left, lower center, and lower rightpoints of the bounding rectangle or measured size of the content. Thevarious types of VDnodes may also define custom-named points that arespecific to the content of that type. For example, if the content is oftype stick figure for representing a stick figure of a person, the stickfigure type may have a custom-named point of “nose” corresponding to theposition of the nose within the bounding rectangle or measured size ofthe stick figure. When the layout engine queries the Lobject for thestick figure coordinates for the named point of “nose,” the Lobjectreturns the coordinates of the nose within the coordinate space of thestick figure.

When an application requests to display a certain anchor point at acertain location on the display, the application provides the anchorpoint and the location to the layout engine. From the anchor point, thelayout engine can identify a path of VDnodes from the root to the anchorVDnode. The layout engine directs the invoking of the Lobjects for theVDnodes along that path. FIG. 22 is a diagram that illustrates anexample visual description tree with a named point. VDtree 2200 includesVDnodes 2201-2214 representing a document with a list as an item withinan outer list. The inner list contains “fraction” items. A fractionincludes a numerator, a line, and a denominator. In this example, thenumerator and denominator are non-numeric images (e.g., a happy face).FIG. 23 is a diagram that illustrates a fraction with a non-numericnumerator and denominator. The fraction 2300 corresponds to VDnode 2206and includes a sad face 2310 as the numerator, a line 2320, and a happyface 2330 as a denominator. The VDnode 2211 is of type happy face andhas a custom-named point of “right eye.” When the Lobject for a happyface is queried for the coordinates for the name point “right eye,” itreturns the coordinates of the right eye of the happy face within thecoordinate space of the happy face. For example, the happy face 2330 mayhave the origin of its coordinate space at the upper left corner 2331 ofits bounding rectangle or measured size. The right eye is located atlocation (50, 50) within the coordinate space of the happy face. Thefraction 2300 considers its origin to be at the center 2321 of line 2320and its coordinates of its bounding rectangle or measured size to be(−80, −170) and (80, 180). The numerator 2310 considers its origin to beat the upper left corner 2311 of its bounding rectangle or measuredsize, the line 2320 considers its origin to be at the center 2321, andthe denominator 2330 considers its origin to be at the upper left corner2331. The bounding rectangle or measured size of the denominator 2310 isa square of size 150 by 150, and the bounding rectangle or measured sizeof the numerator 2330 is a square of size 160 by 160. Vector 2354indicates that the fraction 2300 positioned the denominator 2330 withits upper center point to be at (0, 20) within the coordinate space ofthe fraction 2300. Vector 2355 indicates that the fraction 2300positioned the numerator 2310 with its lower center point to be at (0,−20) within the coordinate space of the fraction 2300. Vector 2351indicates that the origin of the denominator 2330 is at (−80, 20) withinthe coordinate space of the fraction 2300. Vector 2352 indicates thatthe right eye of the denominator 2330 is at (50, 50) within thecoordinate space of the denominator 2330. Vector 2353 indicates that theright eye of the denominator 2330 is at (−30, 70) within the coordinatespace of fraction 2300.

An application may request the layout engine to display the anchor pointthat is the right eye of the happy face at a certain location such as(110, 110) within the display area. FIG. 24 is a diagram thatillustrates an example VLtree generated for the VDtree. The VLtree 2400includes VLnodes 2401-2418. The links between the VLnodes representvisual layout handles that are annotated to show the positioninginformation of the visual layout handles. For example, the positioninginformation for visual layout handle 2423 is (110, 110). The visuallayout handle 2423 indicates to render the origin of VLnode 2403 at(110, 110) within the coordinate space of VLnode 2402. The visual layouthandle 2424 indicates to render the origin of VLnode 2404 at (0, 0)within the coordinate space of VLnode 2403. The visual layout handle2426 indicates to render the origin of VLnode 2406 at (30, −70) withinthe coordinate space of VLnode 2404. The visual layout handle 2431indicates to render the origin of VLnode 2411 at (−80, 20) within thecoordinate space of VLnode 2406. The visual layout handle 2436 indicatesto render the origin of VLnode 2416 at (50, 50) within the coordinatespace of VLnode 2411.

When rendering the VLtree, the rendering engine traverses the VLtreerequesting each VLnode to render its content with its origin at acertain location within the display area. The rendering engine requestsVLnode 2403 to render its content with its origin at (110, 110) withinthe coordinate space of VLnode 2402. The rendering engine requestsVLnode 2404 to render its content with its origin positioned at (110,110), that is, the sum of (110, 110) and the position information (0,0). The rendering engine requests VLnode 2406 to render its content withits origin positioned at (140, 40), that is, the sum of (110, 110) andpositioning information (30, −70). The rendering engine requests VLnode2411 to render its content with its origin position at (50, 50), thatis, the sum of (130, 30) and positioning information (−80, 20).

FIG. 25 is a diagram that illustrates the rendering of the fraction inthe display area. The display area 2500 has the origin of the fractionpositioned at (130, 30), which results in the origin of the denominatorbeing at (50, 50), which results in the right eye being at (110,110)—the requested location of the anchor point within the display.Because of the positioning of the right eye, only a small portion 2501of the numerator is rendered in the display area.

The layout engine generates the VLtree 2400 as follows. The Lobjectcorresponding to the portal VDnode 2202, which is along the path to thenamed point, is invoked along with an indication that the anchor pointis to be at (110, 110) within its coordinate space. Since list VDnode2203 is along the path to the anchor point, the layout engine isdirected to invoke the Lobject for VDnode 2203 passing an indicationthat the anchor point is to be at (100, 100) within the coordinate spaceof the VDnode 2202. The layout engine invokes the Lobject for VDnode2203 passing an indication that the anchor point is to be at (0, 0)within the coordinate space of VDnode 2203. When the invocation for theLobject for VDnode 2203 returns, the layout engine sets the layouthandle for the VLnode 2403 in the VLtree to be positioned at (100, 100),which is within the coordinate space of VLnode 2402.

After invoking the Lobject for VDnode 2203, the layout engine invokesthe Lobject for VDnode 2204 passing an indication that the anchor pointis to be at (0, 0) with the coordinate space of VDnode 2203. When theinvocation for the Lobject for VDnode 2204 returns, the layout enginesets the layout handle for the VLnode 2403 in the VLtree to bepositioned at (0, 0), which is within the coordinates space of VLnode2403.

After invoking the Lobject for VDnode 2204, the layout engine invokesthe Lobject for VDnode 2206 passing an indicating that the anchor pointis to be at (0, 0) within the coordinate space of VDnode 2204. TheLobject VDnode 2206 eventually requests the layout engine to invoke theLobject for VDnode 2211 and positions the origin of VLnode 2411 at (−80,20) within the coordinate space of VLnode 2406. When the invocation forthe Lobject for the VLnode 2406 returns, the layout engine hasdetermined that the anchor point is at (−30, 70) within the coordinatespace of VLnode 2404. The layout engine sets the layout handle forVLnode 2406 to be positioned at (30, −70) within the coordinate space ofVLnode 2404. The net effect of the positioning will be that the anchorpoint is at (0, 0) within the coordinate space of VLnode 2404.

After invoking the Lobject for VDnode 2206, the layout engine eventuallyinvokes the Lobject for VDnode 2211 to generate its VLnodes. Since theVDnode 2211 contains the anchor point, the layout engine also queriesthe Lobject for VDnode 2211 for the position of its right eye within thecoordinate space of VDnode 2211, which is (50, 50). When the invocationof the Lobject for VDnode 2211 returns, the layout engine sets thelayout handle for VLnode 2411 to (−80, 20) within the coordinate spaceof VLnode 2406. The layout engine also determines that the anchor pointis at (−30, 70) within the coordinate space of VLnode 2406, that is, thesum of (−80, 20) and (50, 50). As discussed above, the layout engineuses this position of the anchor point to set the positioninginformation in the layout handle for VLnode 2406 so that the anchorpoint is positioned at (0, 0) within the coordinate space of VLnode2404.

In the example of FIGS. 22-25, the Lobjects for the list VDnodes supportvirtualization, but the other Lobjects do not. When the Lobject forVDnode 2206 returns, it returns the size of its bounding rectangle ormeasured size. The bounding rectangle or measured size of fraction 2300may be represented by the coordinates of (−80, 170) and (80, 180) withinthe coordinate space of fraction 2300 corresponding to the upper leftcorner and the lower right corner. Since the position of the VLnodesresults in the origin of fraction 2300 being positioned at (130, 30)within the display area, the layout engine informs the Lobject forVDnode 2204 that there is no room in the display area above fraction2300, but there is room in the display area below fraction 2300 asillustrated by FIG. 25. Since the Lobject for VDnode 2204 supportsvirtualization, after generating the VLnodes to support rendering theanchor point, it can then request the Lobjects for its other childVDnodes to generate their VLnodes. If there was room at the top of thedisplay area, the Lobject for VDnode 2204 would request invocations forchild VDnodes starting with the VDnode immediately to the left of VDnode2206 and proceeding to the left until the display area is filled to thetop. Since there is room at the bottom of the display area, the Lobjectfor VDnode 2204 requests invocations for child VDnodes starting with theVDnode immediately to the right of VDnode 2206 and proceeding to theright until the display area is filled to the bottom.

In some embodiments, the layout engine may allow an anchor point toanchor the x-coordinate to a named point of one VDnode and they-coordinate to a named point of another VDnode to a specific locationin the display area. In the example of FIG. 22, an application mayspecify that the left side of VDnode 2206 is to be positioned at thex-coordinate of 10 in the display area and that the right eye of VDnode2214 is to be positioned at the y-coordinate of 110. Lobjects mayparticipate in virtualization and take into consideration theapplication's specification when choosing their own coordinate systems.Lobjects may, at their own discretion or under the direction offormatting child nodes, add spacing between children or between childrenand the edge of the allotted display area. The Lobject for VDnode 2202may specify the placement of its child VDnode 2203 in its localcoordinate space such that the left side of VDnode 2206 is positioned atthe x-coordinate of 10 and that the right eye of VDnode 2214 ispositioned at the y-coordinate of 110. The Lobject for VDnode 2203 mayspecify the placement of its child VDnode 2204 in its local coordinatespace such that the left side of VDnode 2205 is positioned at thex-coordinate of 15 (which includes a horizontal margin) and that theright eye of VDnode 2214 is positioned at the y-coordinate of 0 (whichdoes not include a vertical margin).

In some embodiments, a VDtree is annotated to indicate which VDnodeshave been modified so that the VLtree can be regenerates based on theVDnodes that have been modified. A VDnode that has been modified isannotated as being modified, and a VDnode that has not been modified,but has a descendant VDnode that has being modified, is annotated ashaving a modified descendant. When a VDnode is annotated as having beenmodified, then the layout engine regenerates the subtree of the VLtreecorresponding to that VDnode. When a VDnode is annotated as having amodified descendant, the layout engine can determine whether regeneratethe entire subtree of the VLtree corresponding to that VDnode or toregenerate subtrees for certain child VDnodes. The Lobject for theVDnode can decide which child VDnodes need to be regenerated. Even ifthe subtree for a child VDnode need not be regenerated, the layoutengine may need to modify the positioning information of the layouthandle for the child VDnode. For example, if the size of a child VDnodewithin a list is changed, the positioning information in the layouthandle for child VDnodes further down on the list may need to bechanged.

FIG. 26 is a flow diagram that illustrates the processing of a partialregeneration component in some embodiments. The component 2600 is passedan indication of a VDnode and determines whether to regenerate thecorresponding subtree of the VLtree and subtrees for any of its childVDnodes. In decision block 2601, if the VDnode is annotated as beingmodified, the component continues at block 2602, else the componentcontinues at block 2603. In block 2602, the component invokes the layoutcomponent of the Lobject for the VDnode to regenerate the correspondingsubtree in the VLtree and then completes. In decision block 2603, if theVDnode is annotated as having a modified descendant, the componentcontinues at block 2604, else the component completes. In blocks2604-2607, the component loops recursively invoking the component. Inblock 2604, the component selects the next child VDnode. In decisionblock 2605, if all the child VDnodes have already been selected, thenthe component completes, else the component continues at block 2606. Inblock 2606, the component recursively invokes the partial regenerationcomponent passing an indication of the selected child VDnode. Indecision block 2607, if the measured size of the regenerated childVLnode has changed, then the component continues at block 2602 toregenerate the entire corresponding subtree in the VLtree for the parentVDnode, else the component loops to block 2604 to select the next childVDnode. If the measured size of a VLnode changes, then the positioninginformation and measured size for sibling child VLnode may change.

An Lobject may provide behavior that allows additional content to betemporarily added to VDtree. For example, a VDtree may include a VDnodefor a paragraph that is resizable. The Lobject for that type of VDnodemay support the resizing of the paragraph by temporarily adding aVDnode, for each of the corners, representing a graphical user interfacewidget that when selected can be moved to resize the paragraph in eitherone dimension or both dimensions. The layout engine processes thetemporarily added VDnodes by generating corresponding VLnodes for thewidgets. As the widgets are moved, the size of the display area in whichthe paragraph changes and the application regenerates the portions ofthe VDtree affected by the change in size. The layout engine thenregenerates the VLtree, which is then rendered.

In some embodiments, the same content may be displayed in multipledisplay areas simultaneously. For example, a slide presentation may havethe same header displayed on each slide. In such a case, a VDtree forthe presentation may include a VDnode corresponding to the presentationthat has a child VDnode representing the header for the presentation anda child VDnode the non-header content. When multiple slides aredisplayed simultaneously, the header is displayed multiple times. Whenthe header is selected in a displayed slide, the application may providethe behavior of one type of highlighting for the selected portion withinthe selected slide and a different type of highlighting for the selectedportion in each of the other displayed slides. Thus, the hit testingidentifies not only the portion of the header that is selected, but alsothe selected slide. The Lobject for the presentation may generate atemporary VDnode for each of the slides representing the content of theheader. The layout engine will generate a corresponding VLnode for eachof the temporary VDnodes. When the header is selected, the VLnodecorresponding to the header of the selected slide is identified andprovided to the application. The non-temporary VDnode representing theheader may have a child node indicating “primary” and “secondary”selection highlighting or styling, as well as an indication of which ofthe copies is the “primary,” such that the Lobject can produce thecorrectly styled VDnode for each copy.

From the foregoing, it will be appreciated that specific embodiments ofthe invention have been described herein for purposes of illustration,but that various modifications may be made without deviating from thescope of the invention. Accordingly, the invention is not limited exceptas by the appended claims.

We claim:
 1. A method performed by a computing device for generating avisual layout tree for a visual description tree of a document whosecontent is to be fragmented across multiple display areas, the visualdescription tree having visual description nodes and the visual layouttree having visual layout nodes, the method comprising: providing alayout interruption record to identify visual description nodes whoseprocessing has been interrupted; and for each display area that is tocontain content of the document, processing the visual description treeby: setting a current state for processing the visual description nodebased on a state indicated by the layout interruption record for thatvisual description node; populating the visual layout tree with one ormore visual layout nodes representing content of the visual descriptionnode to be included in the display area given the current state; andwhen processing of a visual description node of the visual descriptiontree is interrupted for the display area, adding to the layoutinterruption record for the next display area the then-current state ofprocessing of the visual description node and an identifier of thevisual description node so that for the next display area the processingof the visual description tree can continue from where processing wasinterrupted for the display area; whereby the processing for said eachdisplay area generates a corresponding part of the visual layout tree.2. The method of claim 1 wherein the visual layout tree includes asubtree for each display area.
 3. The method of claim 1 includingperforming a depth first traversal through the visual description nodesidentified by the layout interruption record.
 4. The method of claim 1wherein the layout interruption record for a visual description nodeincludes state information for processing that visual description nodealong with an identifier of a child visual description node and a layoutinterruption record for that child visual description node.
 5. Themethod of claim 1 further comprising caching a visual layout node alongwith state information indicated in the layout interruption record sothat the visual layout node can be reused when subsequently generating avisual layout tree.
 6. The method of claim 1 further comprising for eachdisplay area, displaying within the display area content in accordancewith the one or more visual layout nodes representing content of avisual description node.
 7. A computer-readable storage medium storingcomputer-executable instructions for controlling a computing device togenerate a visual layout tree for a visual description tree whosecontent is to be fragmented across multiple display areas, the visualdescription tree having visual description nodes and the visual layouttree having visual layout nodes, the computer-executable instructionscomprising instructions that: for each of the multiple display areas,traverses visual description nodes of the visual description tree forlaying out content of visual description nodes for the display area asindicated by a layout interruption record storing state associated withvisual description nodes whose layout was interrupted; and generatesvisual layout nodes for the visual layout tree for content of the visualdescription nodes encountered during the traversal of the visualdescription tree based on the layout interruption record associated withvisual description nodes whose layout was interrupted.
 8. Thecomputer-readable storage medium of claim 7 wherein thecomputer-executable instructions further comprise instructions thatgenerate a layout interruption record having state associated with thevisual description nodes whose layout of content was interrupted.
 9. Thecomputer-readable storage medium of claim 7 wherein the visual layouttree includes a subtree for each display area.
 10. The computer-readablestorage medium of claim 7 wherein laying out of content is interruptedas a result of not all of the content of a visual description nodefitting within a display area.
 11. The computer-readable storage mediumof claim 7 wherein the layout interruption record for a visualdescription node includes state information for processing that visualdescription node along with an identifier of a child visual descriptionnode and a layout interruption record for that child visual descriptionnode.
 12. The computer-readable storage medium of claim 11 wherein thelayout interruption record for the visual description node includes anidentifier and a layout interruption record for each child visualdescription node whose layout was interrupted.
 13. The computer-readablestorage medium of claim 7 wherein the computer-executable instructionsfurther comprise instructions that cache a visual layout node along withstate information indicated in the layout interruption record so thatthe visual layout node can be reused when subsequently generating avisual layout tree.
 14. The computer-readable storage medium of claim 7wherein the computer-executable instructions further compriseinstructions that, for each display area, display within the displayarea content in accordance with the one or more visual layout nodesrepresenting content of a visual description node.
 15. A computingsystem for generating a visual layout tree for a visual descriptiontree, the computing system comprising: a memory storingcomputer-executable instructions that when executed: process visualdescription nodes of the visual description tree to layout content ofthe visual description nodes for a display area; and when the laying outof content of a visual description node of the visual description treeis interrupted for the display area, generate a layout interruptionrecord having current state information associated with the visualdescription node whose laying out of content was interrupted so that thelaying out of content of the visual description node for another displayarea can continue based on the state information associated with thevisual description node; whereby the instructions to process the visualdescription nodes of the visual description tree to layout the contentof the visual description nodes for the display area generates thevisual layout tree; and a processor for executing thecomputer-executable instructions stored in the memory.
 16. The computingsystem of claim 15 wherein the visual layout tree includes a subtree foreach display area.
 17. The computing system of claim 15 wherein thelaying out of content is interrupted as a result of not all of thecontent of the visual description node fitting within the display area.18. The computing system of claim 15 wherein the layout interruptionrecord for a visual description node includes state information forprocessing that visual description node along with an identifier of achild visual description node and a layout interruption record for thatchild visual description node.
 19. The computing system of claim 18wherein the layout interruption record for the visual description nodeincludes an identifier and a layout interruption record for each childvisual description node whose layout was interrupted.
 20. The computingsystem of claim 15 wherein the memory further stores computer-executableinstructions that when executed, cache a visual layout node along withstate information indicated in the layout interruption record so thatthe visual layout node can be reused when subsequently generating avisual layout tree.